The present invention relates to a system for processing answers to test questions.
The scoring of test answer sheets involves complex problems. These test answer sheets typically include a series of response positions such as, for example, xe2x80x9cbubbles,xe2x80x9d ovals, or rectangles. A person taking a test would, for example, darken in an appropriate oval with a pencil to answer a multiple choice question. These test answer sheets may also include handwritten answers, such as essay or short answer questions. Systems for scanning and scoring the bubbles on such answer sheets are known in the art. Increased difficulties are encountered, however, when such answer sheets either include other types of answers, such as handwritten answers, or cannot be machine graded. For example, if the student has failed to include his or her name on the test answer sheet, the system may be unable to machine score the test answer.
The goals in scoring test answers that cannot be machine scored include efficiency and consistency. These test answer sheets are typically scored by test resolvers either by manually scoring the physical test answer sheet or scoring an electronic representation of the test answer sheet on a computer. Ideally, the scores provided by the various test resolvers for a particular test question should be consistent, since the scores are used in comparing performance of the students against one another. In addition, a test resolver should ideally work efficiently so as to maintain consistently high scoring rates. The test resolver should not have such a high scoring rate that the consistency or quality of scoring significantly declines; likewise, the test resolver should not have such a low scoring rate that the too few answer sheets are being scored. This manual scoring of test answer sheets, however, makes it difficult to monitor the consistency of scoring among the various test resolvers.
In many situations, test resolvers actually travel to a particular location so that all test resolvers may simultaneously score test answer sheets. Requiring the test resolvers to travel to a given location is inconvenient for the resolvers and expensive for those who administer the tests. Furthermore, tracking the performance of test resolvers against both their own performance and the performance of other resolvers can be very difficult with a manual scoring environment.
The process of resolving test questions is currently done manually, and this presents problems. A resolver is manually presented with the actual test answer sheets for scoring. This process is relatively inefficient, since the resolvers must score the answer sheets one at a time and in the order in which they are presented. Also, manual scoring systems do not have the capability to efficiently gather and categorize the test answers for subsequent analysis. Therefore, with a manual system it is very difficult to determine how teaching methods should be changed to decrease, for example, the number of incorrect answers.
A need thus exists for a system that promotes and achieves consistency and efficiency in scoring or resolving of tests.
The present evaluator management system and method facilitates consistent, accurate, and high quality scoring of data items. A collaborative scoring system and method allows two evaluators to score the same data item and resolves potential discrepancies in the scores provided by the evaluators.
The system also has the capability to use quality items to monitor a resolver""s scoring performance. The system transmits a quality-control test item, having a known score, to a test resolver. After receiving a score from the resolver, the system compares the known score with the resolver""s score and can generate a quality profile for the resolver.
The system also provides performance feedback to test resolvers by monitoring the scoring rates of the resolvers. If a resolver""s scoring rate, in comparison to another scoring rate, contains a discrepancy, the system can report the discrepancy. The system can also recommend breaks to a resolver, or provide a resolver with diversionary activities, in response to a discrepancy in scoring rates. The system also has the capability to route test answers to resolvers based on classifications of resolvers and persons who generated the test answers. Monitoring and selecting resolvers based on classifications promotes fair and consistent scoring of test answers.
Finally, the system provides for an on-line scoring guide to dynamically provide a test resolver with scoring rules for a test answer displayed to the resolver.